Multitenant hosted virtual machine infrastructure

ABSTRACT

A multi-tenant virtual machine infrastructure (MTVMI) allows multiple tenants to independently access and use a plurality of virtual computing resources via the Internet. Within the MTVMI, different tenants may define unique configurations of virtual computing resources and unique rules to govern the use of the virtual computing resources. The MTVMI may be configured to provide valuable services for tenants and users associated with the tenants.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a divisional application of U.S. application Ser.No. 12/434,621, filed May 2, 2009, entitled MULTITENANT HOSTED VIRTUALMACHINE INFRASTRUCTURE, which claims the benefit of each of U.S.Provisional Patent Application No. 61/050,163, filed on May 2, 2008, andU.S. Provisional Patent Application No. 61/101,665, filed on Sep. 30,2008, each of which is hereby incorporated by reference in its entirety.

This application is related to and incorporates in their entirety U.S.Non-Provisional application entitled MULTITENANT HOSTED VIRTUAL MACHINEINFRASTRUCTURE (Attorney Docket No. 64710.8002US08) filed concurrentlyherewith, U.S. application Ser. No. 13/276,153, filed Oct. 18, 2011,U.S. application Ser. No. 13/619,959, filed Sep. 14, 2012, U.S.application Ser. No. 12/434,619, filed May 2, 2009, U.S. applicationSer. No. 12/434,620, filed May 2, 2009, and U.S. application Ser. No.12/434,621, filed May 2, 2009, each of which is hereby incorporated byreference in its entirety.

BACKGROUND

A virtual machine is a software construct designed to run computerprograms like a real physical machine. As an example, a virtual machinemay comprise various software components for running executable codewritten for a particular computing platform. Such executable code mayinclude, among other things, operating system code, applicationprograms, software drivers, and so on.

A virtual machine does not necessarily have a direct correspondence to areal physical machine; however, virtual machines are often designed toimitate the behavior of specific hardware architectures. As an example,a virtual machine may be designed to imitate the behavior of an x86processor.

Virtual machine infrastructures (VMIs) have been developed to coordinateuser access to collections of virtual machines and to provide additionalfeatures such as virtual networking capability, virtual data storagemanagement, and so forth. In general, a VMI comprises a set of physicalcomputational resources, virtualization software for creating virtualmachines on the physical computational resources, and managementsoftware for managing the virtual machines and coordinating access tothe virtual machines.

In one example, a VMI comprises a set of physical computers each runningmultiple virtual machines. Users at separate remote-access consolesaccess the different virtual machines over a local area network. Eachconsole includes a virtual machine interface designed to allow a user tointeract with a virtual machine in the same way that the user wouldinteract with a local machine. For instance, the virtual machineinterface may present a user desktop and explorer windows similar tothose found in an ordinary personal computer.

Within a VMI, different virtual machines may be individually configuredaccording to the users' unique needs and preferences. For instance, thedifferent virtual machines may run different types of operating systems(e.g., Windows, Linux), allowing the users to use differentoperating-system-specific programs within the VMI. Additionally, each ofthe virtual machines may provide a different level of performance sothat the resources of the single physical computer can be efficientlydivided among users having different computational demands.

Recently, large enterprises have begun employing complex VMIs to providevirtual computing resources for large groups of users. These users maywork together, but have different computational demands. As an example,a company with hundreds of employees may set up a virtual data centercomprising many physical machines each configured to run several virtualmachines for use by the employees.

The virtual machines can be configured in accordance with the differentcomputational demands and preferences of the different employees. Forinstance, an employee whose job requires a significant amount ofcomputing power—say, an engineer who runs test simulations on complexcircuits—may use a virtual machine configured with higher throughput andmore memory, compared with an employee whose job only requires the useof a few simple programs such as a word processor and an Internetbrowser. Similarly, an employee whose job requires a relatively highlevel of system access—say, a system administrator—may use a virtualmachine configured with a higher level of access within the VMI comparedwith an employee whose job requires a relatively lower level of access.

Conventional VMIs are designed for use within a single organization,i.e., a single company or group of users. In this type of VMI, theoperation of the virtual machines is governed by a common set of rules,such as a common hierarchy of access rights, usage rights, quality ofservice guarantees, and naming conventions. Additionally, in this typeof VMI, the configuration of resources is controlled by a singleadministrative entity, i.e., a single system administrator or group ofsystem administrators.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual diagram providing a system-level overview of ahosted multitenant virtual machine infrastructure (MTVMI) in accordancewith in some embodiments;

FIG. 2 is a flowchart illustrating the use of a MTVMI by a new tenant insome embodiments.

FIG. 3A is a flowchart illustrating the use of a MTVMI by a userassociated with a particular tenant in some embodiments.

FIG. 3B is a display diagram showing an example browser that could beused to log into a MTVMI, and a resulting screen that may appear in theuser's display upon logging in to the MTVMI in some embodiments.

FIG. 4 is a flowchart illustrating the operation of a MTVMI in relationto a tenant in some embodiments.

FIG. 5 is a flowchart illustrating the operation of a MTVMI in relationto a user in some embodiments.

FIG. 6 is a conceptual diagram providing a more detailed system-leveloverview of a hosted multitenant virtual machine infrastructure in someembodiments.

FIG. 6A is a block diagram illustrating a storage manager in someembodiments.

FIG. 7 is a flowchart illustrating the provision of a software brokeringservice by an MTVMI in some embodiments.

FIG. 8 is a flowchart illustrating the provision of a virtual labservice by an MTVMI in some embodiments.

FIG. 9-25 are display diagrams showing a typical user interfacepresented by an example software application running on a MTVMI toprovide a virtual lab service in some embodiments.

FIGS. 25-26 show additional conceptual diagrams illustrating theorganization of the MTVMI in some embodiments.

FIGS. 27-46 are display diagrams showing displays presented by the MTVMIin some embodiments.

FIGS. 47-49 show a user interface designed for use by vendors offeringitems for sale via the marketplace.

FIG. 50 is a block diagram illustrating a data model representation ofvirtual data centers in the MTVMI management application in someembodiments.

FIG. 51 is a block diagram illustrating component processes in the MTVMImanagement application in some embodiments.

FIG. 52 is a block diagram illustrating the organization of the datamodel that supports access control mechanisms of MTVMI managementapplication in some embodiments.

FIG. 53 is a block diagram illustrating the organization of the datamodel that supports resource usage quota mechanisms of MTVMI managementapplication in some embodiments.

DETAILED DESCRIPTION

A variety of embodiments will now be described. These embodiments areprovided as teaching examples and should not be interpreted to limit thescope of the invention. Although specific details of the embodiments arepresented, these embodiments may be modified by changing, supplementing,or eliminating many of these details.

A shared computing infrastructure (“the infrastructure”) is described.In some embodiments, a user can exploit the entire infrastructure usinga standard web browser, without having to install any specializedapplications. The infrastructure supports the deployment of virtual datacenters, wherein each cluster comprises one or more virtual machinescapable of transparently interfacing with computing resources locatedelsewhere in the Internet. As an example, the infrastructure may executea virtual machine cluster on physical machines within theinfrastructure, while users control the virtual machines throughpersonal computers connected to the infrastructure via the Internet.

The shared computing infrastructure may incorporate a softwareenvironment and interface capable of facilitating the automation ofspecific virtualized infrastructure based services. As an example, theinfrastructure may incorporate a software platform that implements aportal where users can develop and exchange data that may be usefulwithin the infrastructure. This data could include, e.g., virtualnetwork configurations and software applications that could be runwithin the virtual infrastructure. The data could be developed byindividual users or by group collaboration within the infrastructure.Additionally, the data could be exchanged with or without monetarypayments.

For explanation purposes, the shared computing infrastructure will bereferred to as a multi-tenant virtual machine infrastructure (MTVMI). AMTVMI is a VMI adapted for use by multiple tenants, where each “tenant”is an independent entity (e.g., a set of users within a single companyor other organization) whose access to and use of the MTVMI is governedby a unique and independent set of rules. For instance, each tenant in aMTVMI may have a unique and independent authentication mechanism, accesscontrol model, hierarchy of rights and privileges, resource allocationcontrol model, resource usage control, quotas, service level agreements,and billing.

Although the term “tenant” generally has a connotation related to therental or occupation of property or space, “tenant” in the currentcontext should not be limited to entities having certain legal orlogistical agreements or arrangements relative to the MTVMI; neithershould it be limited to entities who occupy a certain quantity of spaceor resources within a MTVMI. Rather, “tenant” here merely refers to anyentity having a certain logical relationship with the MTVMI asdemonstrated by the examples and explanations presented in this writtendescription.

Examples of multiple tenants sharing a MTVMI include different companiesor groups of individual users. For instance, an MTVMI could be shared bymultiple large corporations each having its own cluster of virtualmachines, its own virtual storage management system, and so on.

FIG. 1 is a conceptual diagram providing a system-level overview of ahosted MTVMI 100 in accordance with an embodiment of the invention. Asseen in FIG. 1, MTVMI 100 comprises a physical infrastructure residentat the MTVMI provider's site, virtual infrastructure 105, a softwarelibrary 110, and a MTVMI management application 115.

The left side of FIG. 1 shows different tenants 130 and 140 connected toMTVMI 100 through the Internet 120. Different users among tenants 130and 140 may access MTVMI 100 over corresponding secure Internetconnections. Typically, a user accesses MTVMI 100 by logging on to theinfrastructure through an Internet browser. Once the user logs on toMTVMI 100, a user interface for MTVMI 100 appears in the user's computerdisplay to allow the user to access various features of MTVMI 100. Insome embodiments, the user interface appears similar to a normalcomputer desktop, so that the user's experience is substantially thesame as it would be if the user was interacting with a set of localresources. The user interface enables a user associated with a tenant tomanage virtual machines, such as by creating, configuring, starting, andstopping virtual machines.

In general, tenants can access MTVMI 100 over the Internet from anyInternet-enabled device. As examples, the MTVMI can be accessed throughpersonal computers, Internet-enabled personal digital assistants (PDAs),and Internet-enabled cell phones. In some embodiments, where a useraccesses the MTVMI 100 using a portable device such as a PDA or cellphone, the user does so with a web browser installed in the device, muchlike the user would do in a personal computer.

Each tenant of MTVMI 100 operates within its own unique virtualenvironment. The environment for each tenant may include, for example,one or more unique virtual data centers each comprising a set of virtualmachines, virtual storage, and virtual network accessible to thetenant's users. The environment may further include methods for managingthe cluster, such as a hierarchy of user access rights, limits on theamount of central processing unit (CPU) time and storage space availableto the users, and so on. Additionally, the environment for each tenantmay provide the tenant's users with access to specific softwareapplications found in software library 110, as well as one or more labs112 corresponding to instantiated virtual data centers that can beexecuted.

Within MTVMI 100, virtual infrastructure 105 comprises both a physicalplatform for running virtual machines, virtual storage, and virtualnetwork, and a software platform enabling the creation and management ofthe virtual machines, virtual storage, and virtual network. The physicalplatform may comprise computer hardware such as storage system andmedia, servers comprising CPUs, memory, and related processingcomponents, and networking hardware such as network switches, routers,etc. The software platform may include, e.g., a hypervisor for eachserver, and software components for the storage system and networkingcomponents and software for managing them.

Within virtual infrastructure 105, different functional components canbe roughly partitioned into three groups, including network nodes,virtual machine host nodes, and storage nodes. Network nodes includeboth hardware and software components for implementing virtual networkfunctionality, virtual machine nodes include hardware and softwarecomponents for creating and hosting virtual machines, and storage nodesinclude hardware and software components for implementing virtual datastorage.

The composition and operation of example network nodes, virtual machinehost nodes, and storage nodes is explained further below with referenceto FIG. 6. Although the description treats these different nodes asseparate features, it should be recognized that the functionality ofthese nodes can be implemented by different hardware and softwarearrangements. Moreover, the functionality of the different nodes willnot necessarily be implemented within different physical or logicalstructures. In other words, some components may implement functions formore than one type of node.

Software library 110 comprises a collection of software programs, dataobjects, and/or virtual data center configurations for use by differenttenants of MTVMI 100. The software programs may include, among otherthings, specific operating systems to be run on virtual machines,application programs to run within the virtual machines, and any codeused by tenants to maintain their own virtual resources. The dataobjects may include, for example, database files, software configurationfiles, and so forth. The virtual data center configurations may include,e.g., specifications of network topologies, and hardware and softwarespecifications for individual virtual machines within virtual networks.One example configuration for a virtual machine could include 2 GHzvirtual CPU and 1 GB RAM, and run Windows XP.

Some of the programs and data in software library 110 may be commercialsoftware requiring individual or corporate user-licenses. The licensingof these programs or data may be regulated in any of several differentways. As examples, tenants or individual users could provide their ownlicense keys for the programs, or licenses could be provided as part ofa tenant's agreement for using the MTVMI. In some embodiments, MTVMI 100includes mechanisms for automatically negotiating the licenses withthird party software providers in real time. In some embodiments,licenses are purchased by the entity hosting the MTVMI 100 so that usersare not required to obtain their own licenses.

In some embodiments, software library 110 is divided into public andprivate libraries. A public library is a portion of software library 110accessible to any tenant using MTVMI 100. A private library, on theother hand, is a portion of software library 110 accessible to only asubset of the tenants using MTVMI 100 or to a subset of users belongingto a tenant. Public libraries may include, for instance, freewaresoftware applications, commercial operating systems and applicationsthat can be provided to tenants for a charge based on usage, shareddata, and so on. Private libraries may include, for instance, licensedor proprietary software for use by individual tenants, specific dataobjects related to work performed by the tenant or the configuration ofa tenant's virtual network. In some embodiments, a library can contain acommercial operating system, application, or other program that, wheninstalled on a virtual machine, will be customized with a license keyowned by the installing tenant.

In addition to specific software programs and data structures, softwarelibrary 110 may further include specific virtual data centerconfigurations for use by individual tenants. A virtual data centerconfiguration stored in the software library contains completeinformation for instantiating a set, or “cluster” of virtual machinesconnected by a virtual network. The virtual data center configurationspecifies capabilities of each virtual machine to be instantiated in thecluster, including such attributes as number of processor cores, amountof memory, operating system for which the virtual machine is to beoptimized, etc. The virtual data center configuration further containscomplete state for each virtual machine to be instantiated. Such stategenerally includes contents of a disk volume to be accessible by thevirtual machine, and may also include memory contents of the virtualmachine, and/or processor status for the virtual machine, includingcontents for registers including program counters. In some embodiments,a user may upload a virtual machine image generated outside the MTVMI.This virtual machine image can be stored in the library, andinstantiated by the MTVMI in a lab.

Each of the labs 112 is an instance of a virtual data center created bya tenant's user using the content in the software library. As anexample, one virtual data center may include several virtual machineswith a particular operating system and particular applications, as wellas virtual storage, that is together performing a batch application. Thelabs for a tenant may consist of any number of such configurationinstances, which can be in varying states, including running andstopped.

Further descriptions of software libraries and labs are provided belowwith reference to FIGS. 11-20. Various additional functions and benefitsof software library 110 will be readily apparent from the descriptionsof FIGS. 11-20.

MTVMI management application 115 controls and monitors interactionsbetween tenants and MTVMI 100. Additionally, MTVMI managementapplication 115 allocates resources among different tenants andcoordinates the use of MTVMI 100 by the different tenants. Variousmanagement related tasks performed by MTVMI management application 115may include, as examples, authenticating users, allocating CPU time andstorage space among different tenants, maintaining logical isolationbetween different tenants, tracking each tenant's usage of MTVMI 100,and many others. Additional examples of specific functions that can beperformed by MTVMI management application 115 are described below inrelation to FIG. 6.

Tenant information such as virtual data center configurations and storeddata within MTVMI 100 may be generated in a variety of different ways.For instance, in some embodiments, the tenant information may begenerated by a tenant's privileged user through manual operations on theMTVMI 100 using its graphical interface. Alternatively, virtual datacenter configurations and other tenant information can be generated forMTVMI 100 by providing a machine container that describes an existingphysical or virtual machine or cluster to a module within MTVMI 100, andoperating the module to generate a virtual data center configuration foruse within MTVMI 100. Moreover, virtual data storage facilities can beestablished and initialized within MTVMI 100 by downloading data from anexisting data storage platform into MTVMI 100.

FIG. 2 is a flowchart illustrating the use of a MTVMI by a new tenant insome embodiments. FIG. 2 is intended to provide a very broadillustration of the tenant's interactions with the MTVMI, beginning withthe initial establishment of a usage agreement between the tenant and anMTVMI host. In the description that follows, it can be assumed thatactions performed by the tenant are actually carried out by arepresentative of the tenant, such as a system administrator or anotherauthorized individual.

Referring to FIG. 2, in step 205, a tenant establishes a relationshipwith a MTVMI host by submitting a subscription request to the host. Inthe subscription request, the tenant may specify, among other things, adesired amount of resource usage within the MTVMI, any desired featuressuch as access to particular software applications, and other terms forgoverning tenant's use of the MTVMI. A tenant's resource usage can bemeasured in any of several different ways, including, e.g., by a totalamount of CPU time used by the tenant, or by a total amount of time thatusers are logged on to the MTVMI.

Once a tenant has subscribed to use the MTVMI, the tenant may establishits own virtual data center within the MTVMI. The tenant's virtual datacenter may include, among other things, a set of virtual machines, a setof virtual networks, and a set of data storage resources. Theperformance and capacity of the tenant's virtual data center may belimited by the tenant's subscription agreement; however, the tenant isgenerally free to configure the logical structure of the data center inany way desired. At a high level, the tenant's process of establishing avirtual data center may be compared, by analogy, to the process that anorganization goes through to establish a physical data center, bypurchasing a set of physical resources, connecting the resourcestogether, and getting the resources to communicate with one another.However, in contrast to a physical data center, a virtual data centerrequires significantly less legwork on the part of the systemarchitects, and therefore tends to require much less time and expense.

As part of establishing the virtual data center, in step 210, the tenantestablishes rules to govern use of the virtual data center. Examples ofthese rules include, among others, a hierarchy of access rights andsoftware license privileges for different users, constraints on networktopology or evolution, and limits on the amount of computationalbandwidth and storage space allocated to different users. The accessrights, in addition to limiting use of certain resources by certainusers, may also limit the ability of certain users to change aspects ofthe virtual data center, e.g., by adding additional machines, and soforth. In many regards, the rules governing the virtual data center canbe established so they are similar to the rules that govern ordinary,physical data centers in most organizations. The MTVMI provides theenvironment and tools for the tenant's user to establish thesegovernance rules.

Once rules have been established for the virtual data center, in step215, individual resources of the data center are configured. Forinstance, certain software can be associated with certain virtualmachines, certain users can be assigned as the primary users forparticular machines, and so on. In this regard, step 215 is similar tothe process of installing software and setting up user-accounts in anormal non-virtual data center.

Once resources have been configured for the virtual data center, in step220, the tenant may continue to perform maintenance and monitoring onits virtual data center while the virtual data center is being used bythe tenant's users. The maintenance and monitoring may include upgradingand patching various components, modifying the set of users associatedwith the tenant, measuring the resource usage by different users, and soon.

Those skilled in the art will appreciate that the steps shown in FIG. 2and in each of the flow charts discussed below may be altered in avariety of ways. For example, the order of the steps may be rearranged;some steps may be performed in parallel; shown steps may be omitted, orother steps may be included; etc.

FIG. 3A is a flowchart illustrating the use of a MTVMI by a userassociated with a particular tenant in some embodiments. To use theMTVMI, in step 305, the user opens an Internet browser. As explainedabove, the Internet browser may be included in any type of web-enableddevice, including, e.g., a personal computer, a web-enabled cell phone,or a PDA.

After opening the Internet browser, the user enters a URL for the MTVMIto load an MTVMI access portal website into the browser. In step 310,the user logs onto the MTVMI through the access portal by supplyingcredentials such as a user name and password to the portal.

After the user logs onto the MTVMI, the user has access to the softwarelibrary, labs, and data store. The lab can have multiple configurationinstances, each of which can have multiple virtual machines. The displayof the user's machine may be modified to show a desktop for anyinstantiated virtual machine in the labs. The desktop may resemble anordinary computer desktop as is normally displayed on a personalcomputer. As a result, the user may interact with the virtual machine inessentially the same way that the user would interact with a localmachine.

FIG. 3B is a display diagram showing an example browser 350 that couldbe used to log into a MTVMI, and a resulting screen 355 that may appearin the user's display upon logging in to the MTVMI. As seen in FIG. 3B,the user may have access to a virtual local disk 365, and documents 360stored in a virtual storage platform. In addition to local disk 365, theuser may also have access to other virtual machines within a virtualdata center created by the tenant. In one embodiment, these additionalvirtual machines may be accessible through an icon such as the “MyNetwork Places” icon 370 displayed on the screen 355.

Once the user is logged onto the MTVMI, the user may interact with thevirtual machine in much the same way that the user would interact with anon-virtual local machine. In addition, as will be described in greaterdetail with reference to FIGS. 7-22, a user who has logged onto theMTVMI may also perform actions such as creating a lab with multiplevirtual machines, and installing custom software applications on thedifferent virtual machines for specific tasks.

FIG. 4 is a flowchart illustrating a method of operating a MTVMI inrelation to a tenant in some embodiments. The method of FIG. 4 can beviewed as a counterpart to the method illustrated in FIG. 2. In otherwords, while FIG. 2 illustrates actions performed by a tenant inrelation to the MTVMI, FIG. 4 shows actions performed by the MTVMI inrelation to a tenant. In general, the actions of the MTVMI may beperformed by a MTVMI management application such as that illustrated inFIGS. 1 and 6.

Referring to FIG. 4, in step 405, the MTVMI receives a subscriptionrequest from a new tenant. As described above in relation to FIG. 2, thesubscription request may include, among other things, informationspecifying desired usage rights for the new tenant. In step 410, theMTVMI generates account data for the new tenant based on the informationprovided with the subscription request. The account data may include,e.g., information for regulating MTVMI access and usage by the tenant,such as a data storage quota, authorized users associated with thetenant, a CPU quota, network bandwidth quota, and user authenticationinformation.

After the account data is generated, in step 415, the MTVMI allocatestenant resources, such as data storage space, CPU time, and so forth. Insome embodiments, these resources are allocated on an on-demand basis.In other words, resources such as storage space and CPU time are onlyallotted to a tenant when the tenant performs an operation requiring theresources. In other embodiments, however, resources may be reserved foruse by individual tenants, regardless of whether the tenants currentlyneed resources.

FIG. 5 is a flowchart illustrating the operation of a MTVMI in relationto a user in some embodiments. The method of FIG. 5 can be viewed as acounterpart to the method illustrated in FIG. 3. In other words, whileFIG. 3 illustrates actions performed by a user in relation to the MTVMI,FIG. 5 shows actions performed by the MTVMI in relation to a user.

Referring to FIG. 5, in step 505, the MTVMI receives user login and/orauthentication information. Based on this information, the MTVMIdetermines whether to allow the user to access the MTVMI. Upon grantingthe user access to the MTVMI, in step 510, the MTVMI allows the user toconfigure a lab and any related software. In step 515, the MTVMI allowsthe user to perform authentication of any licenses for selectedsoftware. Alternatively, the MTVMI selects a predefined virtual machineconfiguration for the user. Upon determining virtual resources and anyconfiguration to be used by the user, in step 520, the MTVMI binds theselected predetermined virtual machine configuration to a virtual hostby communicating with the hyperviser for the virtual host. In step 530,the MTVMI begins executing the resources. During the execution ofvirtual machines within MTVMI, the MTVMI may run a console applicationto generate information on the user's display. In particular, the usercan view and interact with the desktop console of the virtual machine ina display such as that shown in FIG. 3B.

FIG. 6 is a conceptual diagram providing a more detailed overview ofhosted MTVMI 100 shown in FIG. 1 in some embodiments. Further to thedescription of FIG. 1, FIG. 6, shows various example components that maybe included in virtual infrastructure 105, software library 110, labs112, and MTVMI management application 115.

In the example of FIG. 6, virtual infrastructure 105 comprises aplurality of network nodes 605, a plurality of virtual machine hostnodes 610, and a plurality of storage nodes 615, each of which can beimplemented using either general purpose commodity server hardware orserver hardware more specialized to the roles of network processing,virtual machine execution, and high-volume reliable storage,respectively.

Each of the network nodes 605 comprises one or more physical machinesthat establish a virtual LAN for each virtual network specified by oneof the labs 112. A virtual network connects together the virtualmachines of a network, and enables them to access their correspondingvirtual data stores on the storage nodes. In particular, to implement avirtual LAN, the network nodes establish an independent set of networkservices for each virtual LAN. These network services include suchnetwork services as virtual private network, firewall, network addresstranslation, dynamic DNS, dynamic host configuration protocol service,and traffic shaping. In some embodiments, one or more of these servicescan be disabled for a particular virtual network. In some embodiments,the separate sets of network services are used to meter the usage ofnetwork bandwidth and/or network services individually for each virtualnetwork. In some embodiments, the user may configure a virtual datacenter to specify whether, for each of these network services, astandard version is executed in the network node, or a specializedversion is executed in the virtual machine. In a similar manner, theuser can configure other networking aspects of the virtual LAN, such asby setting their own static IP addresses which are private only orpublic, connecting individual VPNs from each virtual machine to theirown corporate data center, collectively connecting all the virtualmachines in a configuration to their own corporate data center, settingtheir own domain name, running their own domain name server etc. In someembodiments, one or more of these network nodes work cooperatively as aload-balancing cluster or a failover cluster.

Each of virtual machine host nodes 610 comprises one or more physicalmachines capable of executing one or more virtual machines. Eachphysical machine has a corresponding hypervisor and the one or morevirtual machines runs on top of the hypervisor.

Each of storage nodes 615 comprises one or more physical data storagemedia. These storage media may provide virtual storage for virtualmachines and virtual data clusters running within the MTVMI. In someembodiments, the MTVMI uses load-balancing techniques to select for eachvirtual machine the storage node that is assigned to provide virtualstorage to the virtual machine. In some embodiments, the MTVMI furtherprovides intelligent data placement, dynamic data replication, and/ordynamic data migration. Intelligent data placement involvesstrategically selecting one or more storage nodes in which to storeparticular data. For example, an image for a popular operating systemmay be placed on multiple storage nodes in order to provide for highexpected demand for the operating system among virtual machines. Dynamicdata replication involves creating additional copies of particular dataon different nodes in order to provide better performance for accessingthe data. Dynamic data migration involves moving data to a differentstorage node, such as to relieve a bottleneck. Typically, thesemechanisms are implemented by monitoring and collecting data about thebehavior of host nodes and/or storage nodes and associated workloadlevels, such as which objects are more used than others, and policyintelligence for making determinations based upon this input. In someembodiments, the storage nodes provide virtual network attached storagedevices for use by virtual machines running within the MTVMI.

In some embodiments, the MTVMI uses copy-on-write techniques thatrepresent a child image stored at a later time for particular virtualmachine as a set of changes made to a parent image stored for thatvirtual machine at an earlier time. In such embodiments, a dependencyexists in the child image on the parent image. Where the parent image isto be moved to a different storage node, the MTVMI may migrate the childimage with the parent image, or it may break the dependency of the childimage on the parent image by transforming the representation of thechild image into a complete copy of the child image.

Additionally, storage nodes may provide long and/or short term datastorage for tenant information such as virtual data centerconfigurations, and data archives. In various alternative embodiments,storage nodes 615 may be populated by users or tenants directlyinteracting with MTVMI 100, or through automatic processes such asautomatically downloading content stored in another data storage system.

The Storage Manager component of the VLAB is responsible for allstorage-related operations. The Storage Manager defines a distinct layeron top of the underlying storage products. The purpose of this layer isto present a unified view across these mechanisms, both to centralizestorage knowledge within the application and to insulate the rest of theapplication from the numerous details of the actual storageimplementation.

This layering supports:

-   -   portability across a wide range of lower-level storage        implementations, allowing different COTS storage products to be        deployed on the VLAB as needs determine    -   improved supportability and maintainability    -   optimal use of underlying storage capabilities, as the upper        layer interrogates the lower layers as to capacity/usage, load,        and health status and then delegates specific operations to the        selected node    -   unified management of basic storage operations that can span        multiple or dissimilar underlying storage products—e.g. backing        up, relocating, or replicating data    -   redundant data storage for improved availability    -   horizontal scalability (by adding nodes and agents)    -   resource isolation, for performance, security or availability        considerations—e.g., by isolating a specific tenant's use of        storage to a specific subset of nodes

The Storage Manager is comprised of:

-   -   A collection of “storage nodes” which provide the actual        storage. In some embodiments, these are COTS (Common Off The        Shelf) storage products. There may be several types (with        distinct capabilities) present in the system at any time.    -   A collection of agent processes that run on or near the storage        nodes. Each agent acts as an adapter that supports a common API        and semantics across the different underlying storage systems.    -   A client library that is embedded into client processes. The        library provides local-API access to all storage capability,        manages the overall state of the storage subsystem, acts as a        communication channel to the agents, and collects performance        and load data for the agents.    -   A database and tables that store the state of the storage        subsystem and allows concurrent operations to proceed safely.

Overall, the client processes that embed the client library determinehow to allocate responsibilities to individual storage nodes. Forexample, the client processes determine: for particular data that is tobe stored, on what storage node(s) to store it; when to move particulardata from one storage node to another; when to replicate data on onestorage node to another; and, in response to a retrieval request forparticular data that is stored on each of multiple storage nodes, whichnode to retrieve the data from. In making such determinations, theclient processes consider content, capacity/usage, performance, load,health, and affinity information that the Storage Manager collects andmaintains for each of the storage nodes. In many cases, determinationsmade by client processes involve determining a subset of feasible nodes,then selecting a node from among the feasible nodes.

If new content must be stored that has a dependency on pre-existingcontent (e.g., creating a copy or snapshot of existing content), thenthe nodes that have that pre-existing content are the subset of feasiblenodes. If content must be retrieved, then the subset of nodes that havethat specific content is the feasible subset. In selecting one node fromamong the set of feasible nodes, the client processes seek to avoidnodes that are less healthy (e.g., their disks are in a degraded state)and to prefer nodes that have lower usage and load. Other choices arepossible. In particular, tenants of the service can be assigned anaffinity to a specific subset of nodes (and vice versa where a node isassigned affinity to a subset of tenants), such that requests on behalfof that tenant will target this subset of affinity nodes.

Logic to determine which content should be replicated or migrated and towhich node is also based on the same attributes as above. The logicadditionally schedules these operations during a window when there isless performance impact. In addition, system administrators can manuallytrigger replication/migration of specific data to specific nodes.

Software library 110 comprises software programs for use by tenants andassociated users who interact with MTVMI 100. As examples, softwarelibrary 110 may include a library of commercial and/or private softwareapplications available to a tenant, and a set of software configurationscreated by the tenant or accessible to the tenant. In some embodiments,software library 110 may be connected to a licensing module to negotiateon-demand licensing of proprietary software to tenants, or management ofa limited number of existing licenses among multiple users.

Management application 115 comprises components for managing andmonitoring user access to MTVMI 100. As shown in FIG. 6, for instance,management application 115 may comprise an account management componentfor managing a tenant's account data, a remote console component fordisplaying a graphical user interface in a user's local display, amonitoring and metering component for tracking resource usage bydifferent tenants and users, a backup component for maintaining backupcopies of the data stored by storage nodes 615, an audit logs componentfor maintaining audit logs documenting all the actions performed by eachtenant's users, a network manager component for allowing a tenant toconnect its own physical data center resources to MTVMI 100, a VM hostmanager component for controlling the creation and functioning ofvirtual machines within MTVMI 100, and a storage manager component forcoordinating tenant and user access to storage nodes 615.

In addition to these components, in some embodiments, managementapplication 115 further includes a billing system component formonitoring a tenant's use of resources within MTVMI 100 and producing aninvoice. The monitoring may occur, for example, by tracking the amountof CPU time or data storage consumed by the users associated with eachtenant.

In some embodiments, management application 115 further comprises aresource allocation module for facilitating inclusion of external (i.e.,non-local or other third party) resources into MTVMI 100. In this way,MTVMI 100 can be used to provide potentially brokered access tocomputing, storage and network resources regardless of where theresources are physically located. Such access can be provided accordingto policies for achieving any number of tenant or user objectives, suchas, e.g., cost, reliability, security, availability, and performance.

In some embodiments, management application 115 further includes amodule for specifying a grid configuration of a virtual network based oninput from a tenant. A grid configuration is a specification of thetopology of a set of virtual machines and related networks and storagesystems. The grid configuration module could generate the gridconfiguration by receiving a tenant's description of an existingphysical or virtual network in the form of a tenant-created file or datastructure, and then modifying the description into a form usable byMTVMI 100. The grid configuration can subsequently be imaged onto MTVMI100 for use by particular tenant or user.

Management application 115 may further comprise a security component forallowing tenants to restrict the rights of users to specific physical orvirtual resources, configurations, and other software resources withinMTVMI 100. For instance, when establishing an account for MTVMI 100, atenant could specify a set of users with access to a particular softwaresuite within MTVMI 100, and a set of users without access to thesoftware suite. The security component could then, monitor access to thesuite based on the tenant's specification. Additionally, the securitycomponent could issue alters or perform other actions in response toattempts at illicit access, etc.

In some embodiments, management application 115 further comprises policycomponent allowing tenants to modify resource quotas among individualusers associated with the tenant, or among all users associated with thetenant. More generally, the policy component could allow tenants tomodify any aspect of the tenant's subscription agreement with MTVMI 100.

In some embodiments, management application 115 further comprisesmaintenance components such as patching and virus protection softwarefor updating different the configurations of tenants' different virtualnetworks, backup or replication software for storing redundant copies ofvirtual machines, application packages, installation and testing toolsand services, etc. In general, these maintenance components may operatewith or without input from tenants or users, with or without oversightby the entity hosting MTVMI 100, and in a fully automated,partially-automated, or non-automated manner.

In components allowing tenant or user interactions with MTVMI 100, suchinteractions may take place through a variety of different interfaces,such as graphical interfaces, command line interfaces, programminginterfaces, and so on.

Management application 115 may further include custom monitoring andmanagement components created by tenants or users. To implement thesecustom features, a user could create scripts or application programsusing a scripting program or an application programming interface (API)compatible with MTVMI 100. To facilitate the creation of such programs,management application 115 may additionally include a softwaredevelopment platform for MTVMI 100. By allowing tenants and users tocreate custom software components, MTVMI 100 may provide flexibility andunique capabilities useful to a variety of users having a variety ofneeds.

In addition to the above-described components and features, managementapplication 115 may further comprise software applications for providingspecific virtualization-based services to tenants and users within MTVMI100. Examples of such services will be described below with reference toFIGS. 7 and 8. A more specific example of the service of FIG. 8 willthen be described with reference to FIGS. 9-22.

FIG. 7 is a flowchart illustrating the provision of a software brokeringservice by MTVMI 100 in some embodiments. The service of FIG. 7 isprovided by a software brokering portal where tenants and users areallowed to exchange software and other data that may be useful withinMTVMI 100. The data may include, e.g., custom virtual machine or virtualnetwork configurations, software applications written by independentsoftware vendors, custom scripts or programs written for MTVMI 100,infrastructure models, data models, user rights hierarchy models,process workflows, controls, reports, and financials, to name but a few.

In step 705, MTVMI 100 provides a brokering portal. In one example, thebrokering portal comprises a software application that presents agraphical user interface within a user's display. The graphicalinterface may include a mechanism for presenting available softwareand/or data that can be downloaded by the user. The graphical userinterface may further include a mechanism for allowing the user toupload software to the portal. In some embodiments, the MTVMI grantsaccess to the tenant to software that is already available within theMTVMI. The portal may require the user to provide a form of payment todownload certain software or data. Alternatively, the software or datamay be provided free of charge.

In step 710, MTVMI 100 receives data from the user through the portaland stores the data in a common access area. For explanation purposes, auser who uploads data to the portal is referred to as a seller in FIG.7, even though the portal does not necessarily require users to exchangepayment for sharing data. Similarly, a user who downloads software fromthe portal will be referred to as a buyer.

In step 715, MTVMI 100 presents data to potential buyers within theportal. In step 720, where money transactions are involved in thetransfer of data, MTVMI 100 facilitates those transactions, e.g., byallowing users to debit or credit bank accounts, deposit accounts withinMTVMI, or by adding incurred expenses to a tenant's MTVMI account.

In various embodiments, the MTVMI and the portal together support avariety of uses, including application development and testing; onlineeducation; pre-sales software demonstration; consulting platform; and IPprototyping and operations.

FIG. 8 is a flowchart illustrating the provision of a virtual labservice by an MTVMI in some embodiments. In the example of FIG. 8, theuser creates a virtual lab and executes a software test using thevirtual lab. A virtual lab is a set of virtual computing resourcesconfigured to operate in concert to perform one or more relatedcomputing tasks. As an example, a virtual lab may comprise a set ofvirtual machines and related software configured to process a large dataset such as a genome sequence, or a large video or simulation. A virtuallab may be particularly useful for computing tasks that are performed ona one-time or limited basis, such as software testing, graphicsrendering, and so on. However, virtual labs can also be useful forperforming recurring tasks requiring a significant amount of computingpower. Other example steps that could be performed within a virtual labapplication will be apparent from the description presented below inrelation to FIGS. 9-22.

Referring to FIG. 8, in step 805, a user accesses a virtual labapplication within MTVMI 100. The virtual lab application may generate agraphical user interface on the user's display, such as the interfaceillustrated in FIGS. 9-22. Upon accessing the virtual lab application,in step 810, the user creates or accesses one or more projects. Aproject is a data structure or description that specifies a set ofvirtual resources and a set of users that can access the resourcesspecified by the project. In some embodiments, resource usage isseparately measured and quota compliance is separately enforced for eachproject. By specifying both users and virtual resources, the projectallows specific lab instances to be developed through a collaborativeprocess among the users. Example projects will be described in furtherdetail below with reference to FIGS. 9-10. In some embodiments, atenant's administrative user can specify resource permissions at avariety of different levels of scope, including private scope in whichpermissions are established for a single user, or tenant-wide scope inwhich permissions are established for all users for a particular tenant.Resource permissions can also be specified at all-tenant scope, in whichpermissions are established for users of all tenants. Such permissionsmay relate to items from the software library, the labs, or asset listsestablished by or for tenants.

After establishing the project, in step 815, the user creates a lab byselecting and configuring particular resources within the project. As anexample, the user may select a set of virtual machines from the project,specify particular operating systems for the virtual machines, andidentify software applications to be run on the virtual machines.Examples labs will be described in further detail below with referenceto FIGS. 9 and 11-12.

Once the project and lab have been created, in step 820, the user runsthe lab to perform a particular computational task—in this case, asoftware test. An example of a software test performed in a virtual labwill be presented below with reference to FIGS. 9-22.

FIGS. 9-22 are display diagrams showing a user interface presented by anexample software application running on a MTVMI to provide a virtual labservice in some embodiments. The virtual lab service is availableon-demand over the Internet and allows users to rapidly build andconfigure software test environments using a pre-built library ofvirtual machine images. The virtual lab service further allows users tocreate virtual infrastructure such as virtual processing, networking,and storage facilities on demand using a browser. Finally, the virtuallab service allows users to share data and collaborate with other userswho are also connected to the virtual lab environment via the Internet.

The operation of the virtual lab service will be explained below using aconcrete example involving an engineer uses a MTVMI to test ane-commerce website on behalf of a tenant having users located indifferent parts of the world. The engineer may share components of thevirtual lab with the other users associated with the tenant.

FIG. 9 illustrates a screen that appears after the engineer logs on,through a secure connection, to a MTVMI hosting the virtual lab service.In this regard, the screen of FIG. 9 corresponds to an entry point forthe virtual lab service. As seen in FIG. 9, the screen includes a tablabeled as dashboard 905.

Dashboard 905 includes information related to current projects andrecently used items within the engineer's scope of user privileges. Inparticular, dashboard 905 includes a set of projects 910, softwarelibraries 915, and labs 920 associated with the engineer. The projects910, libraries 915, and labs 920 of FIG. 9 may be created within theillustrated application, or they may have been created previously.

In the current example, we will assume that the engineer will deploy alab that was previously created. To do this, the engineer may simplyselect the lab from the list 920 shown under the heading “My Lab”. As anexample, to select the lab illustrated in FIG. 11, the engineer mayselect a lab 921 entitled “Company A—Build 1340”. Similarly, to selectthe lab illustrated in FIG. 12, the engineer may select a lab 922entitled “Company A—Build 1341”. By selecting one of these labs, theuser can bring up the screen shown in FIG. 11 or FIG. 12, which will bedescribed in further detail below.

In testing the e-commerce site, the engineer verifies the site'sscalability and performance on virtual machines defined in the selectedlab. A realistic test of a e-commerce site typically requires a largenumber of machines. However, the test generally only requires the use ofthe machines for a short period of time, say, a few days or weeks atmost. Accordingly, by performing the test in this virtual lab, theengineer avoids the need to requisition actual hardware and thecorresponding large expense, as would generally be required to test thesite in a non-virtual environment.

In addition to eliminating the need to requisition physical hardware,the virtual lab service also enables the engineer to quickly configureand launch a test environment. In contrast, tests using actual machinesoften require hours of setup time for connecting machines, installingsoftware, booting up the machines, and so on. In selected embodiments ofthe virtual lab service, virtual machines can be requisitioned andlaunched without requesting use of the virtual machines in advance. Thisis typically possible in a MTVMI where the amount of available computingpower is large in relation to the instantaneous user demand.

FIG. 10 shows a projects tab 1005 within the virtual lab service.Projects tab 1005 provides information 1010 regarding a particularproject (“Company A—Security Verification”) accessible to the engineer,virtual machine configurations 1015 available within the project,running configurations 1020 within the project, and assets 1025associated with the project. Each of these features will be described inadditional detail below with reference to the remaining figures.

FIGS. 11 and 12 illustrate different labs available to the engineer. Asseen in the figures, these labs each include a number of virtualmachines. Each virtual machine is visually represented by a thumbnailimage of the console display currently being generated by the virtualmachine. For illustration purposes, each of the virtual machines has abeen configured with a unique combination of computing characteristics,operating system, and status. For instance, in FIG. 11, a web server1105 is implemented by a virtual machine having a 1 GHz CPU, 1 GB ofRAM, and running the Microsoft Windows operating system. Within avirtual network, web server 1105 has a network address “192.168.0.1” anda status “running”. Other virtual machines have differentconfigurations, including different operating systems, differentstatuses, and so forth. Once machines are up and running in theillustration of FIGS. 11 and 12, the engineer can click on a thumbnailimage of any one of the virtual machines to interact with the machinethrough the engineer's local interface.

In the examples of FIGS. 11 and 12, the status of each virtual machine(e.g., running, booting, resuming) can be indicated by both a textualindicator and a background color around the corresponding virtualmachine icon. The virtual lab service may allow users to modify virtualnetwork settings such as domains, subnet attributes and IP addresses ofvirtual machines within a particular lab.

In some embodiments, the virtual lab service enables the user togenerate a snapshot of one or more running virtual machines, such as asingle virtual machine, all of the virtual machines in a virtualnetwork, or all of the machines in a configuration. For example, theuser may select a visual control associated with the machine, network,or configuration in order to generate a snapshot. When a snapshot isgenerated, complete state for each of the virtual machines including thesnapshot is stored in the library. After the snapshot is generated, thisitem may be selected from the library in order to instantiate any numberof new instances of the set of virtual machines that were the subject ofthe snapshot. This is useful, for example, to establish a large numberof initially identical instances of a particular configuration. Also, ina testing project, it can be useful for a tester who has just producedan error in a configuration being tested to establish a snapshot of theconfiguration that can be reviewed later by a developer to determine theconditions under which the error occurred.

In some embodiments, the virtual lab service allows the user to generatefor a particular virtual machine a URL that can be used, in the absenceof any other identifying information, to access the console display forthat virtual machine. This URL can be distributed to users that are notauthorized under the tenant's account to provide access to thisparticular virtual machine to those users. Such access may permit onlyobserving the console display of the virtual machine, or it may permituser interface interactions with the virtual machine. In someembodiments, the user can click on a CD icon displayed by the MTVMI inconnection with a virtual machine (not shown) in order to upload to theasset list and/or the library a virtual machine image or a softwareinstallation file such as an ISO file from the user client machine.

FIG. 13 is a display diagram showing a sample display typicallypresented by the infrastructure in order to permit a user to control theoperation of virtual machines. The display 1300 shows information 1310and 1320 for two virtual machines. The information 1310 for the firstvirtual machine includes information describing the virtual machine, aswell as the following controls: a run/suspend control 1311 for beginningor suspending execution of the virtual machine; a shut down control 1312for invoking an operating system function on the virtual machinedesigned to turn the computer system running the operating system off; apower off control 1313 for immediately disabling the virtual machine anddiscarding its memory state; a RDP connection control 1314 forestablishing an RDP connection with the virtual machine; an ISO control1315 for loading a memory image ISO file from a CD-ROM to the virtualmachine; a delete control 1316 for deleting the virtual machine; and asettings control 1317 for editing network and hardware attributes of thevirtual machine. Each of controls 1311-1317 operate specifically withregard to the virtual machine identified by virtual machine information1310. The display also includes controls 1391-1394 which, when operatedby the user, operate with respect to every virtual machine whose checkbox (e.g., check boxes 1319 and 1329) is checked. The display alsoincludes controls 1381 and 1382 for respectively selecting all of thevirtual machines and deselecting all of the virtual machines.

FIG. 14 is a display diagram showing a sample display typicallypresented by the infrastructure to display information about and allowinteraction with machine configurations available to a user. The display1400 includes entries 1410, 1420, and 1430 each corresponding to adifferent machine configuration that is accessible to the user. In amanner similar to that discussed above in connection with FIG. 13, theseentries contain check boxes 1411, 1421, and 1431, respectively. Each ofthese can be checked so that controls 1491, 1492, 1493, and 1496 operatewith respect to the corresponding virtual machine. The display alsoincludes controls 1494 and 1495 for switching between a list view ofconfigurations and a tiles view of configurations, respectively.

FIG. 15 is a display diagram showing a sample display typicallypresented by the infrastructure to display information about and permitmanipulation of machine configurations in a tiles view. The display 1500includes sections 1510, 1520, and 1530, each containing informationabout a different machine configuration. For example, information 1510about the first machine configuration shows two virtual machines, andincludes controls 1512, 1513, and 1514 whose operation affects each ofthe virtual machines in the first configuration. Information 1510further includes a check box, like check boxes 1521 and 1531 for othermachine configurations, that determines whether controls 1591, 1592,1593, and 1596 apply to the configuration.

FIG. 16 is a display diagram showing a sample display that is typicallypresented by the infrastructure in order to permit the user to specifyautomatic suspension of the virtual machines in a machine configuration.The display 1600 may be reached by selecting an auto-suspend control1602. The display includes information 1610 identifying the machineconfiguration. The infrastructure permits selection between thefollowing options for suspending this machine configuration: an option1621 not to automatically suspend this machine configuration; an option1622 for suspending the machine configuration after all of the virtualmachines in the configuration have been idle for at least auser-specified amount of time; and an option 1623 to suspend the virtualmachines of the machine configuration at a user-specified time and date.The display also includes an update control 1630 for updating the autosuspension settings for this machine configuration.

FIG. 17 is a display diagram showing a sample display typicallypresented by the infrastructure in order to publish URLs for accessingvirtual machines of a machine configuration. The display 1700 can bereached by selecting a Publish VM URLs control 1701. The display enablesthe user to determine with option 1710 whether URLs are generated andactivated for accessing the consoles of the virtual machines of thecurrent machine configuration. Where option 1710 is selected, the usercan elect between option 1711 in which passwords are not required inorder to access a virtual machine using its URL and 1712 where auser-specified password is required to access each virtual machine inthe machine configuration. Also, where the user selects option 1710, theuser can determine whether to select option 1720 for allowing users whoaccess a virtual machine in the machine configuration using itspublished URL to control the virtual machine as contrasted with merelyviewing its console output. Also, where the user selects option 1710,the user may elect between option 1721 to allow those in possession of avirtual machine's URL to access it at all times, and option 1722 toallow access only during a user-specified time-of-day range. The display1700 also includes a save control 1730 to update these attributes of thecurrent machine configuration.

FIGS. 18 through 20 illustrate a library of virtual machineconfigurations available to the engineer within the virtual lab service.The library includes images of configurations that have beenpreinstalled with common operating systems, databases and applications.By creating a lab from pre-installed images, the user can save hours ofmanual installation tasks.

Configurations 1805 shown in FIG. 18 have all been installed in the labof FIG. 11. These and other configurations such as those illustrated inFIG. 19 can be installed in labs or added to currently runningconfigurations by actuating buttons within the interfaces shown in FIGS.18 and 19. Although the engineer may create labs from predefinedconfigurations, the engineer could alternatively assemble and configurenew machine configurations from scratch by specifying a new combinationof virtual hardware, operating system, and so on. The engineer couldthen add the new configuration to an existing lab or create a new labthrough the virtual lab service. In general, any new configurationscreated by a user be associated with that user or shared among otherusers within the MTVMI.

Where a single user or tenant runs multiple instances of a singlenetwork within the virtual lab service, the different instances aregenerally fenced off from each other. Fencing is typically accomplishedby using a virtual private network to establish a separate virtual LANfor each network instance. In some embodiments, the virtual machines ineach of the instances of the network are assigned the same set of IPaddresses. Accordingly, to provide independent access to the Internetfor the virtual machines of each of the instances of the network, thenetwork nodes and/or the firewall proxy requests from these virtualmachines to the Internet, enabling the proxy to route the reply from theInternet to the correct one of the virtual machines having the source IPaddress associated with the original request. In various embodiments,this proxying is performed in a variety of ways, such as mapping eachvirtual machine instance to a different public IP address, URL, orcombination or URL and port number.

Returning to FIG. 11, to test the e-commerce web site, the engineer runsthe web site on the virtual machine designated as web server 1105. Whilerunning the web-site, web server 1105 stores and retrieves data from avirtual machine designated as a database server 1115. Three virtualmachines having a variety of configurations are designated as testclients 1110 for generating traffic for the web-site. Finally, a virtualmachine designated as a test controller 1120 controls test clients 1110to perform the test.

FIG. 21 illustrates techniques that can be used to organize and sharework within the virtual lab application. In particular, FIG. 21illustrates test assets 2105 (i.e., test result data—an asset can be anyproprietary data) that have been generated by tests performed by theengineer within the virtual lab service. Using control 2110, theengineer can upload the assets into a library available to the public orto other users associated with the same tenant. This allows the engineerto share the assets with other users. Once the assets have been uploadedto a library within the virtual lab service, the assets can be added toprojects, as illustrated, e.g., by the information under the “assets”label in FIG. 10. By adding the assets to a project, the assets becomeavailable to other users assigned to the project. Additionally, webbrowsers and other tools executing on a virtual machine may be used to“pull” data and/or programs to a virtual machine from the Internet.

FIG. 22 shows an aspect of the virtual lab application that allows anadministrator to monitor use of the virtual lab service. To monitorspecific types of usage, the administrator may select various parametersfrom drop down menus in an area 2205. For instance, the administratormay select a certain type of activity to monitor, such as CPU usage,memory usage, etc. Alternatively, the administrator may choose tomonitor activity of certain users, activity around certain dates, and soon. Activity that falls within the parameters specified in area 2205 isdisplayed in an area 2210.

FIG. 23 is a display diagram showing a sample display typicallypresented by the infrastructure to show the consumption of resourcesrelative to quotas across an entire customer as well as by an individualuser. The display 2300 includes a section 2310 about consumptionrelative to quotas for the customer, and section 2320 for consumptionversus quotas for the current user. The rows in each section are dividedinto two columns: usage column 2301 that shows the amount that has beenused by the entity and limit column 2302 that shows the quota to whichconsumption is limited for the entity. The customer section 2310includes row 2311 for number of hours that a virtual machine operates;row 2312 for the number of virtual machines allocated; row 2313 for thevolume of persistent storage allocated; and row 2314 for the number ofpublic IP addresses allocated. Section 2320 for the individual userincludes corresponding rows 2321-2323, showing the user's consumptionand the user's limit as contrasted with the customer's total usage andlimit. The display also includes a more concise version 2350 of thisinformation that is suitable to be displayed on a variety of pagescontaining a variety of other information.

FIG. 24 is a display diagram showing a display typically presented bythe infrastructure to report on usage of resources including paidresources accessed via the marketplace. In a list of resources 2310,2360, and 2370, a paid item icon 2311 indicates that resource 2310 ispaid. The entry includes information about the paid item includinginformation 2312 and 2316 identifying the item, information 2313identifying the vendor, information 2314 indicating the basis for whichthe item is metered for payment, and information 2315 indicating thecost of consuming the item.

In some embodiments, a tenant may specify specific levels of service tobe provided by the MTVMI. For example, the tenant may specify that itslabs run in a dedicated virtual machine host node, and/or access adedicated storage node; specify that physical processor utilization beno greater than a level such as 75% on the virtual machine host nodes onwhich its labs execute; or specify a maximum level of resources of oneor more types that the tenant can consume in a particular period oftime. In some embodiments, where quotas are specified for a tenant,either by the tenant or the operator of the MTVMI, the MTVMI displaysusage relative to the quota to one or more of the users for the tenant.

The MTVMI is unique in delivering virtual datacenter Labs as a serviceover the Web based on a multi-tenant shared infrastructure. It provideskey capabilities valued by customers in the initial target markets—QA,Staging, Training, and Pre-sales PoC—including:

-   -   End users can use just an Internet browser to rapidly and easily        create and operate private virtual data center environments        (Labs)—clusters of virtual machines with desired network        architecture and storage resources—without relying on IT        support.    -   Create custom data center configurations that are a near replica        or extension of environments in their internal data center.    -   Multiple concurrent instances of identical configurations can be        cloned and run quickly and easily.    -   Distributed teams can collaborate efficiently and securely.

These capabilities and their underlying technology include:

Multi-Tenant Virtual Lab as a Service with Independent AccountAdministration

-   -   Virtual Lab management application architected as a multi-tenant        service: each “tenant” is a multi-user customer who is provided        authenticated and secure access only to their account and their        virtual resources with no access to the underlying platform or        to other accounts.    -   Account management architecture and data model designed to        support Project Management needs of mature organizations: each        account independently supports multiple users, hierarchy of        roles and administration privileges, organization of users and        Virtual Lab assets in Projects with access control, quota        management, overall account and Project-level resource usage        accounting and auditing.

Software Library Services for Rapid Creation of Custom Configurations(to Emulate Production, for e.g.)

-   -   Multiple methods to build custom virtual machines: upload and        install complete virtual machine images, OS installation        packages on raw virtual machine containers, or application        installation packages on pre-built virtual machine images.

Create and Manage Custom Virtual Data Center in the Cloud

-   -   Create isolated virtual datacenters with wide flexibility in        auto-assigning or user-configuring virtual machine attributes        and identities (e.g., number of cores, amount of RAM, domain        name, IP addresses etc.); policy hooks that enable multi-tenant        aware resource allocation decisions (e.g., for security        isolation, performance isolation etc.).    -   Mechanisms to comprehensively save the state of entire data        center configuration including the virtual machines, their        storage, their network identities, their network service        profiles so that they can be exactly replicated later.

Control Network Configuration of Virtual Data Center and Bridge to OwnInternal Data Centers

-   -   Software-based mechanisms to automatically configure isolated        network services for each virtual data center, including DHCP        and DNS, VPN services and file share services. Ready        connectivity from a virtual data center configuration back to        internal data centers.

Storage Virtualization for Fast and Comprehensive Cloning

-   -   Distributed file system management architecture which supports        fast snapshot and cloning operations in a scalable way,        including data placement, data migration, and load balancing.        The implementation is designed to be hypervisor-agnostic.

Use Commercial Library Items Available in the Library on a Pay-Per-UseBasis

-   -   Architecture, data model, workflows, and metering for supporting        paid library services. Mechanisms for providers to upload        pay-for-use content with the required metadata attributes,        mechanisms for consumers to discover content, and mechanisms for        metering, enforcing, and billing consumers for period of usage.

UI/Console Services for Internet Based Secure Access and Sharing

-   -   Desktop protocol proxy for remote display of single machine and        multi machine environments. Mechanism for controlling desktop        access privileges for various classes of users.

FIGS. 25-26 show additional conceptual diagrams illustrating theorganization of the MTVMI in some embodiments. FIG. 25 shows thephysical arrangement of hardware components of the MTVMI in someembodiments. FIG. 26 shows some of the services provided in connectionwith virtual private local area networks in some embodiments.

FIGS. 27-46 are display diagrams showing displays presented by the MTVMIin some embodiments. FIG. 27 shows a display enumerating availableconfiguration templates that is presented in some embodiments. FIG. 28shows a display presenting information about a single configurationtemplate that is presented in some embodiments. FIG. 29 shows a displayfor monitoring tenant resort resource usage that is presented in someembodiments. FIG. 30 shows a display that a tenant to configure useraccess permissions that is presented in some embodiments. FIG. 31 showsa display that enables a tenant quota to be specified that is presentedin some embodiments. FIG. 32 shows a display for auditing resource usageacross tenants that is presented in some embodiments. FIG. 33 shows adisplay for monitoring tenant resource usage with respect to quotas thatis presented in some embodiments. FIG. 34 shows a display for monitoringthe status of virtual machine host nodes is presented in someembodiments. FIG. 35 shows a display for controlling network settingsfor a virtual local area network is presented in some embodiments. FIG.36 shows a display for reviewing and modifying project settings that ispresented in some embodiments. FIG. 37 shows a display for navigatinginformation associated with a virtual lab service is presented in someembodiments. FIG. 38 shows a display identifying users associated with atenant that is presented in some embodiments. FIGS. 39 and 40 aredisplays showing available virtual machine configurations that ispresented in some embodiments. FIG. 41 shows a display identifyingtenants is presented in some embodiments. FIG. 42 is a displayidentifying users associated with each tenant that is presented in someembodiments. FIG. 43 is a display showing assets available within thevirtual lab service is presented in some embodiments. FIG. 44 is adisplay showing information about a virtual machine configuration ispresented in some embodiments. FIG. 45 shows a display forauthenticating a third-party software license purchased by a tenant ispresented in some embodiments. FIG. 46 shows a display for activating apre-installed instance of an operating system using a license key ownedby, a tenant that is presented in some embodiments. It can be seen thatthe display includes tabs, such as tabs “Windows Server 2003 Standard”and “Windows Server 2003 Enterprise R2 SP2,” that each refer to adifferent virtual machine. The user can of the corresponding virtualmachine.

FIGS. 47-49 show a user interface designed for use by vendors offeringitems for sale via the marketplace. FIG. 47 is a display diagram showinga sample display typically presented by the infrastructure in order toregister a new vendor for the marketplace. The display enables a userassociated with the vendor to enter the following information about thevendor: the company name 4701 of the vendor; the company address 4702 ofthe vendor; the contact name 4703 of a contact at the vendor; the emailaddress 4704 of the contact; a phone number 4705 of the contact; and apassword 4706 to be used by the vendor to access a vendor portal forinteracting with the marketplace. The user can select a submit control4710 to generate a profile that is based upon the inputted information.

FIG. 48 is a display diagram showing a sample display typicallypresented by the infrastructure in order to enable a vendor to list anew item for sale in the marketplace. The display 4800 is typically madeavailable to a user associated with the vendor in response to that usersigning into the vendor portal on behalf of the vendor. The displayincludes information 4801 identifying the vendor. The user can use thedisplay to input the following information about the item to be listed:the name 4811 of the item; a description 4812 of the item; a supportlink 4813 for the item; and a date 4814 on which the listing willexpire. The user may also input pricing for any of a number of differentpricing schemes, including: the per month cost 4821 for each user whouses the item; the per year cost 4822 for each user who uses the item;the per hour charge 4823 for the item; and the one-time charge 4824 peruser for using the item. Typically, if the user does not enter a pricein one of these fields, the option is not offered as a basis on whichbuyers can purchase the item. The display further includes controls 4831and 4832 for specifying a path to the file containing the item that isto be uploaded to the infrastructure from the vendor's computer system.The user can select submit control 4840 in order to create a listing inaccordance with the inputted information.

FIG. 49 is a display diagram showing a sample display typicallypresented by the user in order to account to a vendor for revenuegenerated by the vendor's items in the marketplace. The display 4900shows a period 4901 such as a month, to which the displayed accountingapplies. The user can select controls 4902 or 4903 to navigate todisplay an accounting for other periods of time. The display includes atable in which each row corresponds to a particular basis for chargingcustomers for a particular item. These rows are grouped into group 4920for the Crossbow Debugger item and group 4930 for the Image Cropperitem. Each row is divided into the following columns: an item column4911 identifying the item; a rate column 4912 identifying the ratecharged to users for the payment option to which the row corresponds; aunits column 4913 showing the number of units of that charging optionthat were consumed by customers during the selected period; a revenuecolumn 4914 showing the revenue produced in connection with the currentitem and charging option during the selected period; and an item titletotal column 4915 showing, for the last row of each group, the amount ofrevenue generated by the item during the selected period. The displayalso includes an indication 4940 of the gross revenue for the selectedperiod, and an indication 4950 showing the portion of the revenue thatis retained to support the operation of the marketplace, and anindication 4960 of the net revenue to the vendor for the selectedperiod.

In view of the foregoing, it should be appreciated that, while selectedembodiments have been described herein for illustration purposes,various modifications may be made without deviating from the scope ofthe invention. Accordingly, the invention is not limited except as bythe appended claims and the elements explicitly recited therein.

1. A method in a computing system including at least one processor formanaging virtual storage across a plurality of persistent storagedevices directly or indirectly accessible to processors on which aplurality of virtual machines are executed, comprising: storing data onthe persistent storage devices of the plurality in response topersistent storage requests each from a virtual machine or a virtualmachine management module; autonomously determining to copydistinguished data from a first persistent storage device on theplurality to a second persistent storage device among the plurality; andcopying the distinguished data from the first persistent storage deviceto the second persistent storage device in accordance with theautonomous determination.
 2. The method of claim 1, further comprisingscheduling the copying to occur at a time when persistent storagerequests from virtual machines and the virtual machine management moduleare occurring infrequently.
 3. The method of claim 1, further comprisingdeleting the distinguished data from the first persistent storage deviceafter the copying is complete.
 4. The method of claim 3 wherein theautonomous determination is in response to determining that thedistinguished data has an affinity with data already stored on thesecond persistent storage device.
 5. The method of claim 3 where in theautonomous determination is in response to determining that additionalspace is needed on the first persistent storage device.
 6. The method ofclaim 1, further comprising maintaining the distinguished data on thefirst persistent storage device after the copying is complete.
 7. Themethod of claim 6 wherein the autonomous determination is in response todetermining that the distinguished data is in high demand by the virtualmachines.
 8. The method of claim 1, further comprising: identifying aproper subset of the plurality of persistent storage devices that arecandidates for selection; and selecting a persistent storage device ofthe identified subset as the second persistent storage device.
 9. Themethod of claim 8 wherein a persistent storage device of the identifiedsubset is selected based upon the relative health of the persistentstorage devices of the identified subset.
 10. The method of claim 8wherein the virtual machines are each executed on behalf of one of theplurality of tenants, and wherein at least a portion of the plurality ofpersistent storage devices of the identified subset of been assigned anaffinity for a particular tenant of the plurality of tenants, andwherein a persistent storage device of the identified subset is selectedthat has been assigned an affinity for the tenant on behalf of which thevirtual machine making the persistent storage request is executed. 11.At least one computer-readable medium having contents adapted to cause adata processing device to perform a method for servicing a persistentstorage request, the method comprising: storing data on the persistentstorage devices of the plurality in response to persistent storagerequests each from a virtual machine or a Virtual machine managementmodule; autonomously determining to copy distinguished data from a firstpersistent storage device on the plurality to a second persistentstorage device among the plurality; and copying the distinguished datafrom the first persistent storage device to the second persistentstorage device in accordance with the autonomous determination.
 12. Thecomputer-readable medium of claim 11, further comprising scheduling thecopying to occur at a time when persistent storage requests from virtualmachines and the virtual machine management module are occurringinfrequently.
 13. The computer-readable medium of claim 11, furthercomprising deleting the distinguished data from the first persistentstorage device after the copying is complete.
 14. The computer-readablemedium of claim 13 wherein the autonomous determination is in responseto determining that the distinguished data has an affinity with dataalready stored on the second persistent storage device.
 15. Thecomputer-readable medium of claim 13 where in the autonomousdetermination is in response to determining that additional space isneeded on the first persistent storage device.
 16. The computer-readablemedium of claim 11, further comprising maintaining the distinguisheddata on the first persistent storage device after the copying iscomplete.
 17. The computer-readable medium of claim 16 wherein theautonomous determination is in response to determining that thedistinguished data is in high demand by the virtual machines.
 18. Thecomputer-readable medium of claim 11, further comprising: identifying aproper subset of the plurality of persistent storage devices that arecandidates for selection; and selecting a persistent storage device ofthe identified subset as the second persistent storage device.
 19. Thecomputer-readable medium of claim 18 wherein a persistent storage deviceof the identified subset is selected based upon the relative health ofthe persistent storage devices of the identified subset.
 20. Thecomputer-readable medium of claim 18 wherein the virtual machines areeach executed on behalf of one of the plurality of tenants, and whereinat least a portion of the plurality of persistent storage devices of theidentified subset of been assigned an affinity for a particular tenantof the plurality of tenants, and wherein a persistent storage device ofthe identified subset is selected that has been assigned an affinity forthe tenant on behalf of which the virtual machine making the persistentstorage request is executed.
 21. A system including at least oneprocessor, comprising: a first persistent storage device; a secondpersistent storage device; and a virtual management module configuredto: store data on the first and the second persistent storage devices inresponse to persistent storage requests; autonomously determine to copydistinguished data from the first persistent storage device to thesecond persistent storage device; and copy the distinguished data fromthe first persistent storage device to the second persistent storagedevice in accordance with the autonomous determination.